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Methods for Assessing College Student Use of 
Alcohol and Other Drugs 

by William Dejong, Ph.D. 



Introduction 

If they are to develop effective programs 
and policies to reduce alcohol- and 
other drug-related (AOD) problems on 
campus, college 
and university 
officials must 
understand the 
nature and 
extent of these 
problems at 
their institution. 

Administrators 
can achieve this 
understanding 
only if they 
have accurate 
data on patterns 
of student sub- 
stance use and 
related risk 
behavior. While focus groups, one-on- 
one interviews, and consultations with 
faculty and staff can be helpful, the best 
way to obtain these data is to conduct 
an annual survey using a randomly 
selected sample of student respondents. 

Unfortunately, many campus officials 
are unfamiliar with survey methods and 
do not have the time or staff to develop 
a sound data collection plan. As a 
result, they may avoid conducting rou- 
tine campus surveys or implement them 
in ways that violate basic survey princi- 
ples. Other administrators steer clear of 
student surveys due to fear of negative 
publicity if the findings were to be 
released to the public. 

Avoidance is not an option. Years of liti- 
gation over the concept of “reasonable 
care” in higher education law have shown 
that the courts will typically protect an 



institution that shows good faith by seek- 
ing to apply state-of-the-art prevention 
methods and evaluating their operation 
and effect.^ Currently, the state of the art 
in campus-based prevention is defined 
by the report A Call to Action: 

Changing the 
Culture of 
Drinking at 
U.S. Colleges, 
which reviews 
the research 
literature on 
successful pre- 
vention strate- 
gies and 
provides rec- 
ommendations 
for best prac- 
tices. A key 
recommenda- 
tion in that 
report is to “integrate research into col- 
lege alcohol program planning,” includ- 
ing “data describing the dimensions of 
the college drinking epidemic and its 
effects on students.”^ 

This guide offers a straightforward 
method for gathering and reporting stu- 
dent survey data on substance 
use-related problems. It will be of par- 
ticular interest to program directors for 
AOD prevention programs on campus, 
or to members of a campus-based task 
force or campus and community coali- 
tion that is charged with assessing the 
need for new prevention programs and 
policies. Senior campus officials also 
will find this guide useful for gaining a 
basic understanding of methodology 
for on-campus surveys that focus on 
campus violence and other health and 
safety issues. 



Questionnaire Design 

Good survey instructions, located either 
on the questionnaire’s front page or in a 
cover letter, will introduce the scope 
and purpose of the survey using neutral 
language to avoid biasing how respon- 
dents complete the survey. For example: 
“This survey is about alcohol-related 
attitudes, norms, and behavior. Your 
answers will be used to learn more 
about college students and their percep- 
tions of other students’ alcohol use.” 

The instructions should explain the 
benefits and risks of participation, 
describe any incentives being offered, 
and clarify how the survey will protect 
respondents’ anonymity or confidentiality. 
(Appropriate procedures are explained 
on page 7.) Also, the instructions 
should state that the students’ participa- 
tion is voluntary, and that they can skip 
any questions that make them uncom- 
fortable. Providing contact information 
for the head of the research group is 
another good practice, so that students 
can follow up with any questions or 
concerns they might have. 

A well-designed survey questionnaire 
will ask a wide range of questions to 
determine the scope of AOD use and 
its consequences, identify underlying 
causes, and assess current prevention 
programs and policies. Listed here are 
the major content areas that a survey 
can cover: 

• Personal characteristics that may be 
associated with AOD use, including 
gender, age, racial and ethnic back- 
ground, relationship status, and cur- 
rent employment 



The Drug and Alcohol Abuse Prevention 
Regulations (Education Department 
General Administrative Regulations, Part 
86) require institutions of higher educa- 
tion to conduct a biennial review on the 
effectiveness of their alcohol and other 
drug programs and the consistency with 
which sanctions are applied for policy 
infractions.' A student survey can be an 
important component of this biennial 
review or other evaluations of campus- 
based prevention programs and policies. 



1 



• Family characteristics, including par- 
ents’ education level and substance 
use histories 

• Academic standing, including year 
in school, full- or part-time status, 
grade point average 

• Current residence, including living 
situation (alone, with roommates, 
with family), location of residence 
(on vs. off campus), and type of resi- 
dence (fraternity or sorority house, 
residence hall or dormitory, house or 
apartment, or other) 

• Participation in various student 
activities (e.g., community service, 
religious group, fraternity or sorority, 
intercollegiate athletics) 

• Annual and 30-day prevalence of 
alcohol, tobacco, and other drug use 

• Levels of recent alcohol use (see 
“Example Measures of Alcohol Use”) 

• Alcohol use in specific contexts 
(defined by occasions and settings) 

• Sources of alcohol (e.g., retail out- 
lets, acquaintances) 

• Perceived benefits and risks of alco- 
hol use 

• Consequences due to own alcohol use 
(e.g., hangover, had unprotected sex) 

• Consequences due to other students’ 
alcohol use (e.g., interrupted sleep, 
personal property damaged) 

• Strategies used by the student to 
avoid heavy drinking (e.g., pacing 
alcohol consumption) 

• Strategies used by the student to 
avoid driving after drinking (e.g., 
designated driver) 

• Perceptions of campus drinking norms 

• Awareness and support of campus 
substance abuse policies, local laws, 
and their enforcement (see “Questions 
to Assess Student Support for 
Policy”) 

• Perceptions of the campus and com- 
munity environment (e.g., ease of alco- 
hol access, availability of extracurricular 
and recreational options, level of policy 
and law enforcement, promotion of a 
strong nonuse message for students 
under the minimum legal drinking age) 



Example Measures of Alcohol Use 



A “drink” is defined as “a bottle of beer (12 oz.), a glass of wine (4 oz.), a wine 
cooler (12 oz.), or a shot of liquor (1 oz.) served straight or in a mixed drink. 

30-Day Prevalence 

During the past 30 days, on how many days did you have at least one drink of 
alcohol? 

Weekly Consumption 

What is the average number of drinks you consume in a week? 

Heavy, Episodic Drinking 

Think back over the last two weeks. How many times have you had five or more 
drinks at a sitting? 

Peak Consumption^ 

Think back over the last two weeks. What was the greatest number of drinks 
you consumed at one sitting? For how many hours did you drink? 

a Source: The Core Alcohol and Drug Survey (Long Form), Core Institute, Southern Illinois 
University Carbondale, 111. (see http://www.siu.edu/-coreinst). 

b Source: The Core Alcohol and Drug Survey (Long Form) . The survey should be timed so that it 
does not coincide with a vacation break or the subsequent two-week time period, as students’ 
drinking behavior during vacation may be different than when classes are in session. 

c Source: Social Norms Marketing Research Project, Education Development Center, Inc., 
Newton, Mass, (see http://www2.edc.org/snmrp/survey.pdf). When analyzing the survey data, this 
information can be combined with the respondent’s gender and body weight to estimate the maxi- 
mum blood alcohol concentration (BAC) reached during the drinking episode.^ 



Questions to Assess Student Support for Policy 



To what extent do you support or oppose the following possible policies or pro- 
cedures? [Response alternatives: strongly support, support, oppose, strongly 
oppose] 

Examples: 

• Use stricter disciplinary sanctions for students who engage in alcohol-related 
violence. 

• Apply stricter penalties for the use of false IDs to purchase alcohol illegally. 

• Prohibit kegs on campus. 

• Eliminate low-price bar and liquor store promotions targeted to college stu- 
dents. 



Source: Dejong, W; Towvim, L. G.; and Schneider, S. K. “Support for Alcohol Control Policies 
and Enforcement Strategies Among U.S. College Students at 4-Year Institutions.” /oarTO/ of 
American College Health 56: 231-236, 2007. 
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Questions designed to assess specific pro- 
grams and policies also can be added. 

Several survey instruments are available 
that measure these content areas. It is 
wise to select one of these whenever 
possible, as they have been tested and 
used with other AOD programs: 

• American College Health Association 
(ACHA). ACHA offers the National 
College Health Assessment (NCHA). 
This online instrument broadly 
covers several topic areas: alcohol, 
tobacco, and other drug use; sexual 
health; weight, nutrition, and exer- 
cise; mental health; and personal 
safety and violence. Both paper and 
online survey administration is 
available. Users are able to compare 
their data against the NCHAs 
national survey database (see http:// 
WWW. acha-ncha. org/ index, html) . 
Users are charged a fee. 

• Center for College Health and Safety 
( CCHS). Based at Education 
Development Center, Inc., Newton, 
Mass., CCHS developed the Survey 
of College Alcohol Norms and 
Behavior for the Social Norms 
Marketing Research Project, a 
national research study to evaluate 
the effectiveness of social norms 
marketing in reducing high-risk 
drinking among college students.^ 
The survey, which can be used at no 
charge, includes innovative scales to 
measure student support for policy 
and perceived social capital on cam- 
pus (see http://www2.edc.org/ 
snmrp/survey.pdf). 

• Core Institute at Southern Illinois 
University. The Core Institute mar- 
kets several fee-for-service surveys. 
The best known is the Core Alcohol 
and Drug Survey, which has both a 
long and short form. Both paper and 
online survey administration is avail- 
able. Another popular survey is the 
Campus Survey of Alcohol and 
Other Drug Norms, which includes 
many questions not available in the 
other instrument. The Core Institute 



has also developed a faculty and staff 
survey that may be of interest to 
campus administrators (see http:// 
www.siu. edu/ ^ coreinst) . 

Some campuses administer a general 
health survey one year and an AOD- 
specific survey the next. 

Campus administrators may want to 
develop additional questions to ask about 
issues specific to their campus. Several 
available resources describe the process of 
question development and offer guidance 
on question formatting and phrasing.^ 

Deciding how extensively to cover each 
topic area, while keeping the survey instru- 
ment at a manageable length, can be a 
challenge. Most texts on survey design 
avoid specifying a maximum survey length. 
The standard instruments listed above can 
be completed within 30 minutes by most 
smdents. Within that time frame, the 
longer the survey, the greater the incentives 
needed to get a good response rate. 
Pretesting will be necessary to assess how 
long the survey takes and whether the 
incentives being offered are sufficient. 



Conducting a Survey 

The remainder of this guide outlines how 
to implement and analyze a mail or 
online survey of student AOD use. The 
major steps are: (1) obtaining institu- 
tional review board (IRB) approval, (2) 
selecting the study sample, (3) adminis- 
tering the survey, (4) analyzing the data, 
and (5) reporting the results. Using this 
method, campus administrators will have 
a first-rate study — one that surveys a ran- 
dom sample of students and achieves an 
acceptable response rate. Alternative 
methods for survey administration, dur- 
ing course registration or during classes, 
are also described. 

Obtaining Institutional Review Board 
(IRB) Approval 

In nearly all cases, research involving 
human subjects must be reviewed by 
the college or university’s institutional 
review board to ensure that the necessary 



steps have been taken to protect sub- 
jects from harm and to ensure the 
anonymity or confidentiality of their 
responses. Approval from the IRB is 
needed for both the survey instrument 
and the method of administration. 

If an institution has any federally 
funded research and does not have an 
IRB, one will need to be constituted. 

To find out what is required, contact 
the U.S. Department of Education, 
Office of the Chief Financial Officer 
(see http://www.ed.gov/ about/ offices/ 
list/ocfo/humansub.html) or the U.S. 
Department of Health and Human 
Services, Office for Human Research 
Protections (see http://www.hhs.gov/ohrp). 

Key issues that an IRB will consider 
include whether (1) data will be col- 
lected or stored in a manner that allows 
the identification of any respondents; 

(2) student participation is voluntary; 

(3) any questions may cause emotional 
discomfort or psychological stress, and 

(4) the benefits to participants outweigh 
the risks. Participant benefits might 
include the opportunity to reflect on 
their AOD use, which might lead to 
positive changes in AOD-related 
knowledge, attitudes, and behaviors. 
Note that federal regulations prohibit 
payment or other compensation from 
being considered a participant “benefit” 
when doing a risk or benefit analysis. 
The IRB also will consider benefits of 
the research for the campus, and even 
for society as a whole. 

Selecting the Study Sample 

In conducting student surveys, many 
colleges choose a convenience sample, 
polling only those students who can be 
contacted easily through courses, on- 
campus organizations, or during regis- 
tration. This method is described in a 
later section. 

A preferred method is to select a truly 
random sample. The advantage of 
administering a random-sample survey — 
namely, that the survey results are 
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known to be representative of the stu- 
dent body as a whole — makes the extra 
work involved well worth the effort. 
This section reviews the steps in select- 
ing an appropriate sample. 

The first step is to find a means of 
identifying all potential respondents 
(what researchers call the “sampling 
frame”). In nearly all cases, the institu- 
tion’s official enrollment list, which 
should be available from the college regi- 
strar, can serve this purpose. Having an 
up-to-date student list with accurate 
mailing addresses and telephone num- 
bers is essential. 

The list’s accuracy should be assessed 
after the fact. The Social Norms 
Marketing Research Project, which 
dropped less than 4 percent of its 
national sample, deleted names from 
the sample when (1) there were two or 
more pieces of undeliverable mail; (2) 
telephone contact revealed that the stu- 
dent was not enrolled in the institution, 
had graduated, or was spending the 
semester abroad; (3) the registrar veri- 
fied that nonresponding students 
unreachable by telephone were not 
enrolled after all; or (4) respondents 
indicated on the survey that they were 
not enrolled or were in graduate or pro- 
fessional school.^ 

The second step is to pick a random 
sample of students from the list. When 
an institution has a computer-based 
student records system, administrators 
will be able to use a computer program 
to generate the random sample. 

Absent this capability, the administrator 
can use a random number table to 
select the first name on the enrollment 
list and then select every Ath student 
down the list to be in the sample. With 
this procedure, it is best if the names on 
the enrollment list are randomly 
ordered, rather than alphabetized. The 
size of X will depend on the total 
enrollment and the desired sample size. 
For example, to draw a sample of 500 
students from an enrollment list of 20,000 



undergraduates, administrators would use 
a random number table to pick a student 
from the first 40 names (20,000 divided 
by 500) and then add every 40th name on 
the list to the sample. 

Determining the Sample Size 
How large a random sample should be 
surveyed? There are three deciding fac- 
tors: (1) the level of precision that cam- 
pus administrators want their survey to 
have, (2) the degree of confidence they 
want to have in its results (the esti- 
mates), and (3) the size of the college’s 
student body. 

To illustrate, imagine that researchers at 
a university with 10,000 students have 
received surveys from 964 randomly 
selected students, and that from this 
survey the researchers estimate that 56 
percent of the university’s students can 
be classified as heavy drinkers. It is 
highly unlikely that this estimate is 
exactly correct. More likely, the true 
level of heavy drinking among all 
10,000 students is somewhat higher or 
lower than 56 percent. 

How much higher or lower? It turns 
out that, with a sample size of 964, the 
researchers can be 95 percent confident 
that the true level of heavy drinking is 
within 3 percent of the survey esti- 
mate — that is, somewhere between 53 
and 59 percent. 

This means that there is only a 5 per- 
cent chance that the actual level of 
heavy drinking is either higher or 
lower than this predicted range — that 
is, lower than 53 percent or higher 
than 59 percent. In this scenario, the 
researchers would say that the 95 per- 
cent level of confidence is plus or minus 
3 percent, and that the 95 percent con- 
fidence interval is 53 to 59 percent. 

If the researchers at this institution 
would have been satisfied with a level of 
precision of 5 percent, rather than 3 
percent, then they could have gotten by 
with a sample of only 370 completed 
surveys. In this case, the 95 percent 



confidence interval would have been 5 1 
to 61 percent (i.e., 56 percent, plus or 
minus 5 percent). 

The degree of precision is especially 
critical when comparing the results of 
two surveys. All other things being 
equal, a higher degree of precision (say, 
plus or minus 3 percent, rather than 
plus or minus 5 percent) increases the 
likelihood that a measured difference in 
outcome between two surveys will be 
declared to be statistically significant. 

Return again to the original situation, 
in which the researchers were working 
with a level of precision of 3 percent 
and a range estimate of 53 to 59 per- 
cent. This time, however, imagine that 
the researchers wanted to be 99 per- 
cent confident, rather than 95 percent 
confident, in this range estimate. This 
degree of confidence would require a 
sample of 1,557 completed surveys. In 
this case, there would be only a 1 per- 
cent chance that the actual level of 
heavy drinking is either higher or 
lower than the predicted range (i.e., 
the 99 percent confidence interval) of 
53 to 59 percent. 

Table 1 shows the relationship between 
the level of precision specified for a 
percentage estimate (e.g., plus or 
minus 3 percent), the confidence limit 
(95 percent or 99 percent), the size of 
the college’s student body, and the 
required sample size of completed sur- 
veys. What is immediately obvious 
from the table is that as the level of 
precision moves from 5 to 3 percent, 
and as the confidence limit moves 
from 95 to 99 percent, the required 
sample size increases dramatically. 

As can be seen in the table, in general, 
administrators should secure completed 
surveys from a sample of between 400 
and 1,000 students. If the college’s 
budget permits it, a sample of 1,000 is 
far superior. Even under the severest 
budget constraints, the sample should 
not be any smaller than 200. 



4 



Methods for Assessing 



TABLE 1. Random sample sizes needed to achieve various degrees of 
precision for percentage estimates from student surveys 



Degree of Precision® 


Student Body Size 


Confidence Limit^ 






95% 


99% 


3% (d = .03) 


500 


341 


393 




1,000 


516 


649 




2,000 


696'' 


959 




5,000 


800 


1,347 




10,000 


964 


1,557 




20,000 


1,013 


1,688 


5% (d = .05) 


500 


217 


285 




1,000 


278 


399 




2,000 


322 


498 




5,000 


357 


586 




10,000 


370 


622 




20,000 


377 


642 



a The degree of precision, also known as the margin of error, is the amount of tolerated error (e.g., plus or minus 3 
percent) in a percentage estimate from a student survey (e.g., 48 percent of the students are heavy drinkers). 

b The confidence limit is the relative degree of certainty with which a percentage estimate is made (e.g., 95 percent 
confidence). 

c Example: For a percentage estimate that is accurate to within plus or minus 3 percent, and to be 95 percent confi- 
dent in that estimate, a sample size of 696 is required if the total population size is 2,000. In this case, a finding 
from the student survey might be expressed as follows; “An estimated 48 percent of the students are heavy drinkers, 
with a 95 percent confidence limit of 45—51 percent.” 



This table assumes that the estimated heavy drinking rate from the student survey will be 50 percent — or, equiva- 
lently, that the proportion estimate p will be 0.50. If there is no information about what the proportion is likely to 
be, then 0.50 is the most conservative estimate, as it dictates the largest possible sample size for the specified degree 
of precision and the chosen confidence limit. If a college has conducted a prior survey, then its value for the propor- 
tion estimate ip) can be used in the formula shown below to calculate what sample size is needed in the next survey. 



This formula can also be used when an institution’s actual student body size differs significantly from the 



precise values shown in the table: 






n = 



1 + 



N V, d' ) 



Where: n = the sample size needed; t = 1.960 for a 95% confidence limit, or 2.576 for a 99% confidence 
limit; p = the proportion estimate (e.g., .50); q = 1-p; d = desired degree of precision (e.g., for 3%, d = .03); 
and N = the student body size. 
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In drawing the sample, allow for the 
anticipated response rate. As outlined 
below, there are several steps that can be 
taken to increase the percentage of stu- 
dents who complete the survey, but it is 
unlikely that the response rate will 
exceed 50 to 60 percent. Also allow for 
inaccuracies in the college’s enrollment 
list; a good rule of thumb is to add up 
to 20 percent to the desired sample size. 

To illustrate, imagine that administra- 
tors wanted a final usable sample of 
1,000 respondents. Assuming a 50 per- 
cent response rate, they would draw an 
initial sample of 2,000 students. Then, 
to account for errors in the enrollment 
list, they would increase the initial sam- 
ple by 20 percent to 2,400 students. 

Thinking along these lines, campus 
officials who expect a response rate well 
below 50 percent may be tempted to 
compensate by administering the survey 
to all or a very large number of stu- 
dents. Actually, to produce a truly rep- 
resentative sample, it would be far 
better to select a smaller random sample 
and then offer incentives and do inten- 
sive follow-up work to get a higher 
response rate. 

How large a sample is drawn also 
depends on the kinds of comparisons 
that administrators might want to make 
among subgroups of students. The 
more complicated the analysis, the 
larger the sample needs to be. To illus- 
trate why this is so, imagine that 
administrators surveyed a sample of 400 
students, 50 percent of whom are male. 
Imagine further that 50 percent of the 
students, both males and females, are 
black and that 50 percent are white. 
This means that one-fourth of the stu- 
dents are black men, one-fourth are 
white men, and so forth. The size of 
each subgroup is only 100 students. 

This means that percentage estimates 
for any one of these subgroups would 
have a level of precision of only about 
10 percent. If these four subgroups 



were subdivided even further — say by 
fraternity or sorority membership — 
then the level of precision would be 
even less. 

A good rule of thumb is to avoid sub- 
dividing the sample into categories 
beyond the point at which the number 
of students in any one of the resulting 
subgroups falls below 50. On the other 
hand, if comparing complexly defined 
subgroups is an important part of the 
study, then drawing a random sample 
of more than 1,000 students will avoid 
the pitfall of having subgroups that are 
too small. 

Finally, administrators always have the 
option of giving the survey to all of 
their students. Whether this is practical 
depends on the 
size of the stu- 
dent body and 
the institution’s 
capacity to 
administer the 
survey in a cost- 
effective way. 

For example, if 
the college has 
walk-through 
course registra- 
tion, students 
could be asked 
to complete a 
survey as part 
of that proce- 
dure, with due 
care taken to 
achieve high response rates while also 
maintaining the anonymity or confiden- 
tiality of student responses. Administrators 
at colleges with 1,000 or fewer stu- 
dents should give this option serious 
consideration. 

Administering the Survey; Mail and 
Online Surveys 

There are two main methods of survey 
administration; by mail and online. 
Until recently, most surveys were done 
by mail, but more and more colleges are 



switching over to Internet-based sur- 
veys. Such surveys offer considerable 
advantages, including cost savings, espe- 
cially for large student samples; faster 
response times; improved survey design 
features such as forced-choice fields, 
programmed skip patterns, and interac- 
tive data validation; and automatic data 
entry into a database, ready for statisti- 
cal analysis. 

The available evidence suggests that 
mail and online surveys yield data of 
comparable quality. In one randomized 
control study, the results showed no sta- 
tistically significant differences in stu- 
dent demographics, response rates, or 
item completion errors between mail 
and online surveys of health risk behavi- 
ors.^ Another investigation found that 
reliability and 
validity assess- 
ments of alco- 
hol use 
measures did 
not vary by 
online vs. tra- 
ditional paper- 
based 
methods. 

Internet-based 
surveys are 
particularly 
appropriate for 
a college stu- 
dent popula- 
tion, the vast 
majority of 
whom use the Internet and e-mail. A 
2002 study found that 86 percent of 
college students had gone online and 
that 72 percent check e-mail at least 
once daily." A growing number of col- 
leges use e-mail and the Internet in 
course work," and many require stu- 
dents to check their campus e-mail 
accounts regularly for institutional com- 
munications. Some colleges and universi- 
ties are unable to do online surveys 
because too few students use their college 



Telephone surveys of college students are 
not recommended. First, an adequate 
sampling frame is usually not available, 
because the registrar’s office seldom has a 
record of student cell phone numbers. 
Second, due to frequent telemarketing 
calls, many people, including students, 
let voice-mail or an answering machine 
take their calls, or they use caller ID to 
screen out calls from unfamiliar num- 
bers. And, once reached, people are now 
more likely to refuse to participate in 
telephone surveys than was the case sev- 
eral years ago.® 
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e-mail address, and campus officials 
have not yet required students to report 
their operative address. 

Obtaining Informed Consent, With a 
mail survey, students implicitly grant 
their consent to participate in the study 
after reading an introduction that 
explains the study’s purpose, the volun- 
tary and confidential nature of the sur- 
vey, measures designed to protect 
anonymity or confidentiality, the set of 
offered incentives, the benefits and risks 
of participation, and contact informa- 
tion for the lead researcher. The intro- 
duction also makes clear that they can 
skip questions without penalty. 

For online surveys, students are sent an 
e-mail that includes a link for the sur- 
vey’s Web site address. When students 
click on the link, they should first see 
the study introduction. To encourage 
students to read this material, there can 
be a series of check boxes that the stu- 
dents click on as they read each para- 
graph. At the end, there should be a 
button that students click to indicate that 
they have read the material, understand 
it, and are willing to participate in the 
survey under the outlined conditions. 

Note that student responses are 
exported to a database only after the 
student clicks a “next” button at the 
end of each survey page, meaning that 
they may choose to terminate their par- 
ticipation at any time during the survey. 

Maximizing the Response Rate. A key 
concern for both mail and online sur- 
veys is the possibility of a low response 
rate, which would compromise the use- 
fulness of the data. Strictly speaking, 
failing to obtain data from any student 
who is part of the random sample 
would make the data less representative 
of the entire student body. At one time, 
researchers conducting student surveys 
would look to obtain response rates of 
at least 70 percent. Recently, however, 
national mail surveys have achieved rates 



only between 50 and 60 percent, even 
with incentives and multiple reminder 
messages.'^ While some researchers have 
reported low response rates with Internet- 
based surveys, others report achieving 
rates comparable to or even somewhat 
higher than mailed surveys. 

A key to achieving a response rate of 50 
percent or more is to inform students 
about the steps being taken to ensure 
their anonymity or confidentiality. 
Research has established that self-report 
surveys about substance use are generally 
both valid and reliable, but only under 
conditions that convince students that 
their responses cannot be linked to them 
by anyone outside the research team.'^ 

A paper-and-pencil survey can be com- 
pleted anonymously. There should be 
no ID number or other identifying 
information on the questionnaire or, in 
the case of a mail survey, the return 
mailing envelope. Also, the instructions 
should remind students not to put their 
name on the questionnaire. For a mail 
survey, students can be asked to mail in 
a separate, self-addressed postcard with 
a unique ID number when they have 
completed the survey so that researchers 
can keep track of which students need 
to be sent future reminders. In practice, 
very few students return the postcard 
but not the survey. 

An online survey can be completed 
under conditions that maintain confi- 
dentiality, but not anonymity. It requires 
a more elaborate procedure. Each stu- 
dent selected to be in the study should 
be assigned a unique password ID, 
which must be entered to gain access to 
the online questionnaire. This is neces- 
sary so that students who are not 
selected cannot access the site, and to 
block multiple responses from the same 
student. If students begin the survey but 
do not complete it, they should be able 
to log in again, but their prior responses 
should not appear on the screen, thereby 
protecting students from the possibility 



of someone else finding their password 
ID and accessing their responses. 

Students should be informed that only 
their password ID, not their names or 
e-mail addresses, will be directly linked 
to their individual responses. The fol- 
lowing steps can be taken to maintain 
students’ confidentiality: (1) Encrypt all 
data so that it cannot be accessed by 
anyone other than a member of the 
research team. (2) Ensure that the data 
file will be available only to a predefined 
subset of research staff with password- 
guarded access. (3) Assign user privi- 
leges to other members of the research 
team that allow them only to view 
aggregate data, but not individual sur- 
vey responses. (4) Maintain students’ 
names and their unique password IDs 
in one database, while keeping the sur- 
vey data, organized by student password 
ID, in a separate database with strictly 
controlled access. With this procedure, 
student names will never be directly 
linked to student data in any database, 
and only a small set of authorized users 
will have password-guarded access to 
both databases. 

Offering incentives is another key strategy 
for increasing the response rate.''’ For a 
mailed survey, the Social Norms 
Marketing Research Project included a $ 1 
bill with the first survey mailing as an up- 
front incentive.'^ Students completing the 
survey, as indicated by a separately mailed 
return postcard with their ID number, 
became eligible for three prize drawings: a 
$100 cash prize per institution for stu- 
dents responding within one week; five 
$50 cash prizes per institution for stu- 
dents responding by the end of the semes- 
ter; and a $1,000 national grand prize for 
students responding by the end of the 
semester.'® Smaller cash awards (or even 
prizes such as T-shirts, store coupons, and 
so forth) might work for a survey done on 
a single campus. Students completing an 
online survey can receive store discounts 
or coupons online. 
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An important step is to send reminder 
messages to encourage students to com- 
plete the survey. Without follow-up, a 
random sample survey of students will 
typically have a 30 percent response 
rate. The Social Norms Marketing 
Research Project used the following 
schedule of mailings to boost response 
rates for its annual mail survey at 1 8 
colleges and universities: (1) “teaser” 
postcard announcing the survey two to 
three days before the first survey mail- 
ing; (2) first survey mailing, sent about 
three weeks after the start of each insti- 
tution’s semester; (3) reminder postcard 
sent to nonrespondents; (4) second sur- 
vey mailing, sent to arrive two weeks 
before spring break; (5) reminder post- 
card sent to remaining nonrespondents; 
(6) third survey mailing, sent about 
three weeks after spring break; and (7) 
fourth survey mailing, with a two-page 
version of the survey. Research staff 
even made reminder telephone calls 
after the second and third survey mail- 
ings; for each round, the caller left a 
voice message on the third attempt. 

Reminders for online surveys are simpler 
to execute. Some experts recommend 
sending a sequence of reminders, begin- 
ning three days after the initial e-mail 
notification and continuing episodically 
for one month.^“ The number of follow- 
up reminders can vary, depending on the 
speed with which the desired response 
rate is obtained. 

Researchers have identified still other 
measures to boost response rates. As 
already noted, the survey introduction 
should inform students that their par- 
ticipation is voluntary and that they do 
not have to answer any question that 
makes them feel uncomfortable. It may 
also help to tell students that they are 
part of a small, randomly chosen group, 
and to ask for their prompt response, 
noting that this would help ensure that 
the survey results truly represent the 
student body. Finally, it is important to 
tell students that the college’s IRB is 
requiring the researchers to conduct the 



survey in a manner that will protect the 
students’ anonymity or confidentiality; 
describe the measures in place to meet 
that requirement; and provide the 
name, telephone number, and e-mail 
address of the IRB coordinator who can 
answer any questions they might have. 

Alternative Survey Methods 
The survey method just presented, 
which involves selecting a random sam- 
ple of students and then surveying them 
by mail or online, is the best procedure 
for obtaining findings that are truly rep- 
resentative of the student body as a 
whole (unless, of course, it is possible to 
survey the entire student body). Unfor- 
tunately, not all colleges and universities 
are in a position to administer their sur- 
veys in this way. 

The most common barrier that institu- 
tions encounter is the time and money 
it takes to mail out the survey and then 
to do the necessary follow-up by mail 
or telephone to get a high response rate. 
And it must be emphasized that getting 
that high response rate is essential to 
the success of this method. Without it, 
administering the survey in this way 
may hold no special advantages over 
other, more convenient methods. 

Two alternative methods are described 
here: administering the survey during 
course registration or during classes. In 
practice, neither of these methods can 
result in a true random sample being 
selected, but they can produce response 
rates of 70 percent or even higher. 

To make these alternative methods 
acceptable, steps must be taken to ensure 
that the student sample is selected in 
such a way that it is very similar to the 
student body as a whole. How that can 
be done is outlined below. 

It is important to stress, however, that 
such a sample — even one that can be 
demonstrated to be very similar to the 
entire student body on characteristics 
like gender, race and ethnicity, year in 
college, and so forth — will not be truly 



representative of that student body. Only 
a random sample can be described that 
way. These alternative methods can be 
made acceptable, but they remain sec- 
ond-best options and should be recog- 
nized as such. 

Administering the Survey During 
Course Registration. One alternative 
method for administering the survey is 
to integrate the data collection process 
into other campus routines, such as 
course registration. This will work only 
if all (or nearly all) students are required 
to report to a central location during 
some part of the registration procedure. 

Registration officials will need to be 
contacted in advance to work out 
arrangements. Ideally, approaching stu- 
dents about participating will be built 
in as a step in the registration process. 
Space will be needed for up to 10 stu- 
dents at a time to complete the survey 
in relative privacy. Staff provided by the 
survey team should be available to dis- 
tribute and collect the survey forms and 
to answer questions. 

To make sure that students know that 
the information they provide will 
remain anonymous, they should receive 
a memorandum that explains the pur- 
pose of the survey and how the com- 
pleted questionnaires will be collected 
and stored. Each student should be pro- 
vided with an unmarked envelope in 
which to put the completed survey. 

With a small enough student body, it 
might be possible to have every student 
complete the survey. Institutions with 
larger enrollments will need to select 
every Ath student in line. For example, 
with a student enrollment of 20,000, 
every 25th student could be approached 
to generate a sample size of 800 students. 

Selecting every Axh student could result 
in a random sample, if all students were 
required to come to a central location 
to register, and if the selection proce- 
dure were followed perfectly. That is 
unlikely to happen, however. If the 
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research staff are able to cover registra- 
tion only during certain time periods, 
or if a heavy flow of students makes it 
physically impossible to ensure that 
every Xth one is selected, a nonrandom 
sample will result. 

Participation should not be required. 
Even so, high rates of participation can 
be built by making an effective personal 
appeal, giving a careful explanation of 
the procedures, and providing extra 
incentives, such as snacks, coupons, or 
eligibility for prize drawings. 

Administering the Survey During 
Classes. A second alternative method for 
administering the survey is to distribute 
it during classes. This method is inex- 
pensive. The only significant costs are for 
the test administrators, who might 
instead be recruited as volunteers. This 
method is also quick. Surveys can usually 
be administered within a week. 

Again, selecting a convenience sample of 
this sort is not recommended as a first 
choice, because as usually practiced, it 
does not result in a sample that is truly 
representative of the student body as a 
whole. Even so, administering the survey 
during classes may be the only method 
that colleges and universities without 
adequate resources to conduct a random 
sample survey will be able to use. 

It should be noted that there are condi- 
tions under which this procedure can 
generate a true random sample. 
Specifically, there must be a class (or a 
type of class) that each and every stu- 
dent is required to take at the same 
time, such as freshman English or phys- 
ical education. Ideally, each class section 
should have approximately the same 
number of students. Weighting proce- 
dures can be applied to correct for hav- 
ing class sections of different sizes; 
consult a statistician for additional 
information. 

In administering the survey, the 
researchers draw a random sample of 
class sections and then ask every student 



in those sections to complete the instru- 
ment. By this method, each student has 
an equal probability of being selected 
into the sample, which is the defining 
characteristic of a random sample. The 
statistical method for calculating “confi- 
dence intervals” is different for this type 
of sample compared with a simple ran- 
dom draw of students; 
consult a statistician for 
additional information. 

At most colleges, how- 
ever, these conditions 
cannot be met, and 
therefore a true ran- 
dom sample of stu- 
dents cannot be 
drawn using this pro- 
cedure. At best, even 
if the researchers 
take great care, the 
sample will be only 
an approximation of 
a random sample. 

Does this matter? 

Yes. The problem 
is that selecting a 
nonrandom sam- 
ple of students 
can lead to inac- 
curate results. 

The same type of 
inaccuracy can 
creep in if classes 
are selected by convenience rather than 
at random. If only English or mathe- 
matics classes were chosen, the source 
of the expected bias would be obvious. 
If only small classes were chosen, there 
would be a different type of bias. The 
point is that, no matter how the classes 
are chosen, if they are chosen non- 
randomly, some kind of bias can be 
expected. 

The amount of bias can be controlled 
somewhat. By selecting the right classes, 
researchers can draw a student sample 
that matches the total student body in 
several important respects (e.g., gender, 
age, race and ethnicity, year in college). 



Doing this is a major advantage. Even 
then, however, researchers must accept the 
fact that this student sample would not be 
truly representative of the student body as 
a whole. This is not said to discourage 
campus officials from doing a survey, but 
to remind them of the real limitations pre- 
sented by this survey method. 



Unfortunately, not 
every institution of higher education 
will have the technical capability to iden- 
tify a set of classes that will result in a 
good sample. The college’s computer 
records must include key demographic 
information about its students, and there 
must be accurate, up-to-date records of 
which classes each student is taking. 
Moreover, all of this information must 
be stored in a database that can be easily 
manipulated to identify the best classes 
to select. 

There also is the difficulty of getting fac- 
ulty to grant permission for the survey to 
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be conducted in their classes. Many 
instructors will cooperate, but some 
may be reluctant to set aside the time. 
Researchers have found it helpful to 
solicit senior administrator support, 
explaining the importance of surveying 
all randomly selected classrooms and 
the benefits of collecting the data. 
Faculty may also respond to an appeal 
that acknowledges the students' loss of 
instruction time but also underscores 
the benefits to the entire campus com- 
munity from the knowledge acquired 
through the survey. Even so, researchers 
may not get access to all of the classes, 
which must be taken into account 
when judging whether the resulting stu- 
dent sample is representative. 

With classroom administration, mem- 
bers of the research team, not class 
instructors, should administer the sur- 
vey. This should be done at the begin- 
ning of the class period to make sure 
there is enough time. As with other 
administration procedures, due care 
must be taken to ensure that students 
know that their participation is volun- 
tary and that their responses will remain 
anonymous. This message can be 
underscored by not walking around the 
classroom as the surveys are being com- 
pleted, reminding students to respect 
one another’s privacy, and collecting the 
completed surveys in a single box or 
envelope. 



Analyzing the Data 

Most campus administrators will need 
to do only two types of data analysis to 
get a good picture of student drinking 
at their institution: descriptive statistics 
and tests of association. Administrators 
interested in more sophisticated data 
analyses (e.g., significance tests, multi- 
ple regression) will need to consult with 
statisticians — possibly those on their 
own faculty. 

Descriptive statistics are used to summa- 
rize the students’ survey responses. Such 



statistics are used to answer questions 
such as the following: (1) What per- 
centage of students at the college can be 
classified as heavy drinkers? (2) On 
average, how many drinks per week do 
students consume? 

Counting how many students fall into a 
category produces a statistic known as 
the frequency. K frequency table itemizes 



the number of students in each of sev- 
eral categories. The percentage of stu- 
dents in each category can also be 
calculated, based on either the total 
number of responses or the total num- 
ber of valid responses (that is, with 
respondents with missing information 
excluded). Frequency data are often 
reported using pie diagrams or bar 
graphs. The results for question 1 
above could be presented this way. 

A useful way to summarize a data set 
involving rating scales or other numeri- 
cal scores is to report the average or 
typical measurement, what statisticians 



call the data set’s central tendency. 

There are three measures of central 
tendency: mode, median, and mean. 
The mode is the score that occurs most 
frequently. Note that a data set can 
have more than one mode if the maxi- 
mum frequency is shared by two or 
more scores. The median is the score 
that separates the upper half of the 
scores from the 
lower half That 
is, 50 percent of 
the scores are 
larger than the 
median, and 50 
percent are 
smaller. The mean 
is what most peo- 
ple think of as the 
average, lo cal- 
culate the mean, 
add up all the 
scores and divide 
that sum by the 
total number of 
scores. 

Tests of association 
such as the fr 
(chi-square) test 
or Pearson’s product 
-moment correla- 
tion coefficient 
are used to assess 
the existence of a 
relationship 
between two variables. For example, 
administrators might want to know 
whether there is a significant relation- 
ship between how many drinks stu- 
dents usually consume when they 
drink and the amount they drank the 
last time they rode with a designated 
driver. Tests of association also can be 
used to examine whether two or more 
groups have truly different frequency 
distributions for some measurement. 
For example, campus administrators 
might ask whether a greater number of 
fraternity members are frequent heavy 
drinkers than nonmembers. 



Rates of heavy, episodic drinking among college stu- 
dents vary greatly across U.S. colleges and universities. 

A seminal survey of 140 four-year colleges found that 
the percentage of students who drank heavily ranged 
from 1 percent at the college with the lowest incidence 
to 70 percent at the college with the highest.^' 

One reason for this wide variation is that the student 
bodies at these colleges are different, and certain types 
of students are more likely to be heavy drinkers than 
others. “ For example, white students are several times 
more likely to be heavy drinkers than students of color. 
Thus, all other things being equal, colleges with a 
higher percentage of white students would be expected 
to have a greater number of heavy drinkers. Other criti- 
cal factors include gender, age, marital status, residential 
status, and membership in fraternities or sororities. 

The implication is clear: An individual college should not 
compare its heavy drinking rates with overall national 
averages without taking into account the nature of the 
institution and its student body. 
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Reporting the Results 

Student surveys are used to determine 
the scope of AOD use and its conse- 
quences, ascertain the individual and 
environmental factors that contribute to 
the problem, and assess the reach and 
effectiveness of prevention programs 
and policies. Baseline surveys are a criti- 
cal part of a problem analysis, the first 
step in a systematic strategic planning 
process. Later surveys are used to 
monitor progress in meeting program 
objectives and reducing AOD-related 
problems. 

Many college administrators are reluctant 
to report publicly the results of their insti- 
mtions student surveys, even when there 
has been progress in reducing AOD- 
related problems. As noted before, collect- 
ing and using survey data to develop, 
monitor, and refine prevention program- 
ming is consistent with best practice in the 
field. Given that essential purpose, it is 
counterproductive to hold back on pub- 
licly reporting any survey results or other 
research findings. 

The major concern, of course, is that 
the survey findings will bring negative 
attention to the institution. It must be 
remembered, however, that the public 
already knows about college students’ 
involvement in substance abuse, espe- 
cially illegal and excessive alcohol con- 
sumption. In short, this is not news. 
Also, administrators should consider 
that in most cases the data will show 
that the problem is less severe than the 
public imagines, based on the news 
reports, movies, and television shows they 
see. Admitting that a minority of stu- 
dents attending the college are part of a 
national problem is no disgrace. Rather, 
reporting and discussing these data are 
the first steps in bringing focused atten- 
tion to the problem and building support 
for evidence-based programs and policies. 
Even if later surveys show that progress 
is slow or remains elusive, the need for 



public disclosure remains just as com- 
pelling. Confronting reality, not hiding 
from it, is the key to galvanizing a stronger 
commitment to address the problem. 

When instimtions are making progress in 
combating substance use, a failure to 
report survey findings is not just counter- 
productive, but self-defeating. A college 
that is widely known as a “party school” 
will continue to attract high-risk smdents 
who are looking for that kind of experi- 
ence, and misperceptions of actual student 
drinking norms will continue to perpetuate 
the problem. If the college’s reputation is to 
catch up with reality, then good informa- 
tion about its efforts to confront the prob- 
lem and any encouraging survey infor- 
mation should be proclaimed far and wide. 



Conclusion 

Ideally, college administrators will commit 
themselves to a long-term program of 
research so that they can monitor changes 
in student alcohol use and other campus 
conditions. This is essential for assessing 
the effect of an institution’s prevention 
programs and policies. The student survey 
methods outlined in this guide should be 
a key component of that research effort. 

This guide has shown that the best survey 
method involves seleaing a random sample 
of smdents and then sending them a survey 
by mail or online, with a strong follow-up 
effort made to generate a high response 
rate. Due to limited resources, not all insti- 
tutions are in a position to administer their 
surveys in this way, and they must consider 
second-best alternative methods, such as 
administering surveys to classes. As dis- 
cussed, these methods can be conducted in 
a way to minimize their limitations and 
produce usable, if not ideal, results. 

Given the need for reliable and accurate 
information, and given the clear superior- 
ity of studying a true random sample of 
students, college officials should consider 
increasing their research budget to make it 
possible to use the preferred survey 



method. The relatively modest resources 
that are needed are not excessive, especially 
considering the improved quality of infor- 
mation that would result. Ultimately, 
good decision-making must rest on a 
foundation of accurate information. 

William Dejong, Ph.D., is a professor of 
social and behavioral sciences at the 
Boston University School of Public Health 
and a senior adviser to the Higher 
Education Center for Alcohol and Other 
Drug Abuse and Violence Prevention. 
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Sources of Information 



Office of Safe and Drug-Free Schools 
(OSDFS) 

U.S. Department of Education 
http:// www.ed.gov/ osdfs; 202-245-7 896 
OSDFS supports efforts to create safe 
schools, respond to crises, prevent alcohol 
and other drug abuse, ensure the health and 
well-being of students, and teach students 
good character and citizenship. The agency 
provides financial assistance for drug abuse 
and violence prevention programs and activ- 
ities that promote the health and well-being 
of students in elementary and secondary 
schools and institutions of higher education. 

The U.S. Department of Education’s 
Higher Education Center for Alcohol and 
Other Drug Abuse and Violence Prevention 

h ttp : // WWW. highe redcenter.org; 1-800- 
676-1730; TDD Relay-friendly, Dial 711 
The Higher Education Center considers 
strategic planning and evaluation to be an 
important component of a comprehensive 
prevention approach. The Higher Education 



Center has several publications and other 
materials to help campus administrators 
develop and evaluate prevention programs. 
These materials (examples listed below) can 
be accessed for free from the Publications and 
Evaluation sections of the Center’s Web site. 

The Network Addressing Collegiate 
Alcohol and Other Drug Issues 

http://www.thenetwork.ws; see Web site 
for telephone contacts by region 
The Network Addressing Collegiate 
Alcohol and Other Drug Issues (Network) 
is a national consortium of colleges and uni- 
versities formed to promote healthy campus 
environments by addressing issues related to 
alcohol and other drugs. Developed in 1987 
by the U.S. Department of Education, the 
Network comprises member institutions 
that voluntarily agree to work toward a set of 
standards aimed at reducing AOD problems 
at colleges and universities. 



Publications 

College Alcohol Risk Assessment Guide: 
Environmental Approaches to Prevention 

by B. E. Ryan; T. Colthurst; and L. Segars 
This guide outlines methods for iden- 
tifying and analyzing factors in the 
campus and community environment 
that contribute to alcohol-related 
problems (104 pp., 1997). 

How to Select a Program Evaluator 

by L. Langford and W. Dejong 
This document describes the skills, expert- 
ise, and experience to look for when seek- 
ing an evaluator; questions to ask when 
assessing an evaluator’s past work; and 
guidance on how to network to find the 
right person and forge an effective work- 
ing relationship (3 pp., 2001). 

Evaluating Environmental 
JVlanagement Approaches to Alcohol 
and Other Drug Abuse Prevention 

by W. Dejong and L. M. Langford 
This document outlines the basic steps 
for evaluating a program for alcohol 
and other drug abuse prevention that 
features environmental change efforts 

(6 pp., 2006). 
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review.] 

5 . College Alcohol Risk Assessment Guide: Environmental 
Approaches to Prevention (CARA) 

6. Methods for Assessing Student Use of Alcohol and Other 
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for Alcohol and Other Drug Abuse 
and Violence Prevention 



Our Mission 

The mission of the U.S. Department of 
Educations Higher Education Center for 
Alcohol and Other Drug Abuse and Violence 
Prevention is to assist institutions of higher 
education in developing, implementing, and 
evaluating alcohol, other drug, and violence 
prevention policies and programs that will foster 
students’ academic and social development and 
promote campus and community safety. 



How We Can Help 

The U.S. Department of Educations Higher Education Center offers an integrated array of 
services to help people at colleges and universities adopt effective prevention strategies: 

• Resources, referrals, and consultations 

• Training and professional development activities 

• Publication and dissemination of prevention materials 

• Assessment, evaluation, and analysis activities 

• Web site featuring online resources, news, and information 

• Support for the Network Addressing Collegiate Alcohol and Other Drug Issues 



Get in Touch 

Additional information can be obtained by contacting: 

The Higher Education Center for 

Alcohol and Other Drug Abuse and Violence Prevention 

Education Development Center, Inc. 

55 Chapel Street 
Newton, MA 02458-1060 

Web site: http://www.higheredcenter.org 

Phone: 1-800-676-1730; TDD Relay-friendly, Dial 711 

E-mail: HigherEdCtr@edc.org 
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